PiPNN 6/6: add HashPrune candidate merging by SeliMeli · Pull Request #1295 · microsoft/DiskANN

weiyaoluo (SeliMeli) · 2026-07-29T16:41:41Z

Overlapping leaves can emit many duplicate or directionally redundant candidates for one source. The direct PiPNN path retains every unique candidate until final RobustPrune. This PR adds optional HashPrune merging: each source streams candidates into a bounded reservoir keyed by residual direction, keeping the nearer candidate when hashes collide. Direct merging remains the default.

Concepts

lsh.rs projects each point onto seeded random hyperplanes. For edge source → target, signs of target/source projection differences form a relative hash. Similar residual directions collide.

l_max is logical per-point reservoir capacity. With final_prune=true, extraction returns up to l_max candidates to shared RobustPrune. With false, extraction returns the nearest graph-degree candidates directly. HashPrune bounds candidate merging; it does not own graph degree, metric, or prune policy.

Code map

diskann/src/graph/pipnn/mod.rs
- HashPruneConfig validates structural bounds and effective capacity against graph degree.
- PiPNNBuildContext::with_hash_prune opts in and checks effective capacity against graph degree.
- build_graph_inner performs the build-wide architecture/metric selection; build_graph_for<A, M, T> preserves direct mode and selects the two consuming extraction paths without redispatch.
lsh.rs creates deterministic random-hyperplane sketches with per-worker conversion scratch.
hash_prune.rs
- hot metadata slots plus padded cold hash/distance/neighbor slabs;
- one raw mutex per source row;
- empty, same-hash, not-full, early-reject, and eviction insertion branches;
- full-candidate and nearest-only consuming extraction.
leaf_build.rs converts symmetric leaf output to deduplicated directed CSR, gathers active sketches, translates local/global IDs, and streams edges into reservoirs.
bf16.rs supplies compact ordered distance storage.
diskann-wide owns dispatched relative-hash/hash-scan implementations; PiPNN never names an ISA.
Disk and benchmark configuration thread the optional policy through existing adapters. Missing hash_prune remains direct mode; an explicit nested object opts in.

End-to-end flow

Dataset → one build-wide architecture/metric selection → seeded point sketches + per-point reservoirs → unchanged overlapping partitioning → leaf-local nearest pairs using the same concrete A, M → deduplicated directed CSR → gather active leaf sketches → compute relative hash per edge → lock only source reservoir → insert/replace/reject → consume reservoirs:

final_prune=true: full candidate lists → private shared finalization/RobustPrune → degree-bounded graph.
final_prune=false: nearest R candidates → degree-bounded graph.

Without with_hash_prune, direct candidate merging is unchanged.

Invariants and boundaries

min(l_max, 2^num_hash_planes) must cover graph degree. Disk builders enforce this before quantizer training or artifact creation.
scan_lanes is physical SIMD padding; l_max is logical capacity. Padded cells are never candidates.
Each hot slot owns one source lock. Length, farthest cache, and cold row mutate only under that lock.
A hash appears at most once per logical row; same-hash replacement requires a closer total key.
(bf16 distance, residual hash, neighbor ID) gives history-independent retention under ties.
Leaf-local IDs become global dataset IDs before reservoir insertion.
Grow-only sketch scratch is read only through the active prefix; empty-edge ingestion preserves existing scratch.
Consuming extraction releases unused sketch/hash/distance storage before or while IDs materialize.
Raw-pointer kernels require padded valid ranges and exclusive source mutation; strict-provenance Miri covers them.
Windows slab FFI uses Rust 2024 unsafe extern; Linux mmap and Windows VirtualAlloc preserve zero-backed lazy allocation semantics.

Review path

Start with HashPruneConfig, with_hash_prune, and orchestration branches.
Review slab allocation/drop and Send/Sync safety invariants.
Establish the single-writer rule in with_locked; trace every insertion mutation and farthest-cache update.
Compare scalar references with diskann-wide dispatch, including padded tails and signed-zero/NaN buckets.
Trace one leaf through CSR dedup, sketch gather, ID translation, insertion, and both extraction paths.
Finish with disk/benchmark configuration.

Test architecture

Tests are grouped by source conversion, dispatched hash primitives, slab/configuration, leaf ingestion/scratch, reservoir replacement/order, and concurrency/extraction. Weak or duplicate cases were removed only when replaced by exact stronger oracles:

seeded serial LSH references replace probabilistic seed inequality;
exact reservoir victim/list assertions replace length-only eviction tests;
one distinct-direction fixture proves full extraction and exact nearest truncation;
parallel insertion compares exact serial neighbor lists;
repeated full builds compare canonical neighbor sets, not unspecified row order;
empty-edge scratch reuse and finalization overflow propagation remain explicit.

Names describe behavior without mechanical test_/should_ prefixes. Private and composition tests are co-located in the implementation files that own each seam.

Validation

All 439 diskann library tests pass with pipnn,testing, including HashPrune configuration, graph composition, runtime fanout, and contention cases.
Six production disk-adapter tests and the dedicated benchmark lifecycle test pass, including fail-fast capacity validation before quantizer artifacts.
Strict-provenance Miri passes relative-hash, padded hash-scan, full-reservoir eviction, and contended hot-slot mutation cases.
Windows GNU and AArch64 cross-target checks pass.
No standalone PiPNN crate or PiPNN microbenchmark target is submitted; the local regression harness remains outside the PR stack.

Stack relation

Stack 6/6. Depends on #1294 and completes the series. Private RobustPrune (#1315), numerical kernels (#1287), core construction (#1290), disk serialization (#1291), and benchmark lifecycle (#1294) remain the owning layers for those concerns.

Stack 6/6: #1294

weiyaoluo (SeliMeli) · 2026-07-29T17:28:35Z

Azure BigANN10M validation (Standard_D16as_v5, AVX2, 16 threads, 4× interleaved per binary) used R=64, c512/c64, fanout=[10,3], k=2, l_max=72, hp=12, final_prune=true.

Build: 76.777s clean → 79.601s stack (+3.68%)
Peak RSS: 12.224 → 11.709 GiB (-4.22%)
Recall@10 L=50: 0.957825 → 0.957790 (-0.0035pp)
Mean comparisons/hops L=50: +0.023% / +0.006%

Raw runs: ~/pipnn-pr5-ab/results/final-head-k2-20260729T171245Z on the benchmark VM. The residual build gap is being investigated only on local, unpushed pipnn-stack/07-perf-investigation; current attribution points to repeated per-row FP16 conversion dispatch and CSR bookkeeping, not graph quality or HashPrune semantics.

weiyaoluo (SeliMeli) · 2026-07-30T08:48:53Z

Direct final-stack QC

Reviewed directly against current origin/main; no subagents were used.

Fixed during QC

Removed the proposed PiPNN-owned unsafe FP16 gather because its measured ~0.48% gain did not meet the unsafe RFC's safety/CI-evidence bar.
Added PR6's missing direct bytemuck dependency (caught by a clean coverage build).
Updated shared RobustPrune for the latest Neighbor accessor API.
Rebuilt the stack from exact per-layer commit boundaries and rebased it onto current main.
Preserved current-main nightly feature coverage while retaining strict HashPrune Miri jobs.

Remaining findings

Evidence gap: performance-motivated unsafe kernels have Criterion/Azure/Miri evidence, but no CI-integrated performance regression threshold as required by rfcs/00109-unsafe-rust.md.
Performance objective not met: allocator-matched BigANN10M 4× interleaved A/B reports build median 75.899s clean vs 77.473s final (+2.07%); wall +1.62%; RSS -3.60%; recall effectively unchanged (0.957805 vs 0.957795). No parity/win claim is made.

Validation

Formatting and CI-style workspace Clippy: pass.
PiPNN all targets: pass (81 unit tests plus integration/config/kernel suites and benchmark smoke executables).
RobustPrune/Vamana tests: 22 pass.
Focused disk PiPNN tests: 5 pass.
pipnn and disk-index,pipnn feature compilation/Clippy: pass.
x86-64 baseline and AArch64 all-target checks: pass.
Strict-provenance Miri HashPrune kernel checks: pass.
PiPNN ISA/TLS/RefCell/broadcast-cleanup scans: zero hits.
Stable production-source coverage: 94.19% lines / 91.88% regions; experimental branch coverage 91.32%.
Complexity/duplication audit: no maintainability blocker.
Final Azure BigANN10M smoke: Recall@10 L=50 95.774%, build 77.918s, peak RSS 11.68 GiB.

Local runtime benchmark integration remains blocked by unavailable Git LFS (the fixture is a 123-byte pointer); CI's LFS checkout is the remaining runtime oracle.

Full report: /tmp/pipnn-qc-direct.md in the authoring environment.

weiyaoluo (SeliMeli) · 2026-07-30T09:12:18Z

QC follow-up

The first post-rebase CI run exposed a real AArch64-only failure in cosine_zero_norm_masks_nan_norm_at_simd_boundaries: NEON and x86 have different max(NaN) behavior, so a NaN pair could become distance zero on ARM. PR1 now preserves NaNs explicitly with a validity mask before selecting the non-negative clamp.

The replacement run is green on:

Ubuntu AArch64 runtime tests
x86-64 Nehalem/SDE baseline
AVX-512/SDE
Ubuntu and Windows default/all-feature tests
coverage, CodeQL, formatting, and all Clippy matrices

Remote stack metadata was also rebuilt as one stack, #1301:
#1287 → #1288 → #1290 → #1291 → #1294 → #1295.
The obsolete closed HashPrune PR is no longer a stack member.

Copilot

Pull request overview

Adds an optional HashPrune/LSH-based candidate-merging path to PiPNN builds, wiring it through disk build configuration and benchmarks, and extending SIMD/mask utilities needed by the new kernels.

Changes:

Introduces HashPrune reservoirs plus random-hyperplane LSH sketch computation, and integrates them into PiPNN leaf building / extraction (optionally followed by RobustPrune).
Extends disk-build and benchmark pipelines to accept and validate HashPrune parameters for PiPNN.
Adds supporting utilities (trusted adjacency-list constructor, mask helpers, SIMD eq optimization) and CI Miri coverage for the raw-pointer kernels.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
diskann/src/graph/adjacencylist.rs	Adds `from_vec_trusted` for zero-copy construction when uniqueness is guaranteed.
diskann-wide/src/doubled.rs	Adds `first()` support for doubled masks.
diskann-wide/src/arch/x86_64/v3/i16x16_.rs	Optimizes SIMD equality mask generation using movemask+pext.
diskann-vector/src/lib.rs	Makes x86_64 prefetch helpers available without requiring AVX2.
diskann-pipnn/tests/config.rs	Adds validation tests for `HashPruneConfig` parameters.
diskann-pipnn/tests/build_graph.rs	Adds parallel HashPrune build invariant test.
diskann-pipnn/src/lsh.rs	New LSH sketch computation (seeded random hyperplanes) and errors.
diskann-pipnn/src/lib.rs	Adds HashPrune config/types and integrates candidate-merge selection into `build_graph`.
diskann-pipnn/src/leaf_build/tests.rs	Adds CSR construction tests used by HashPrune leaf streaming.
diskann-pipnn/src/leaf_build.rs	Refactors leaf computation; adds CSR edge streaming into HashPrune reservoirs.
diskann-pipnn/src/hash_prune.rs	New HashPrune implementation (hot/cold slabs, per-row locking, SIMD hash ops, extraction).
diskann-pipnn/src/hash_prune/tests.rs	Adds unit and concurrency tests for HashPrune kernels and reservoir behavior.
diskann-pipnn/src/bf16.rs	Adds bf16 packing helpers for compact distance keys.
diskann-pipnn/Cargo.toml	Adds new dependencies and a HashPrune benchmark target.
diskann-pipnn/benches/hash_prune.rs	Adds criterion benchmark comparing direct vs HashPrune merge paths.
diskann-disk/src/lib.rs	Re-exports `HashPruneParameters` when pipnn feature is enabled.
diskann-disk/src/build/mod.rs	Re-exports `HashPruneParameters` from configuration.
diskann-disk/src/build/configuration/mod.rs	Exposes HashPrune parameters in configuration module exports.
diskann-disk/src/build/configuration/build_algorithm.rs	Extends `PiPNNParameters` with `hash_prune` and serde defaults.
diskann-disk/src/build/configuration/disk_index_build_parameter.rs	Switches to returning borrowed PiPNN parameters for build selection.
diskann-disk/src/build/builder/build/pipnn.rs	Wires optional HashPrune parameters into `PiPNNBuildContext`.
diskann-disk/src/build/builder/build/pipnn/tests.rs	Updates PiPNN disk builder tests for new parameter passing.
diskann-disk/src/build/builder/build.rs	Validates PiPNN + HashPrune config when pipnn is selected.
diskann-benchmark/src/index/build.rs	Wires optional HashPrune parameters into benchmark PiPNN builds.
Cargo.lock	Adds new transitive dependencies for diskann-pipnn changes.
.github/workflows/nightly.yml	Improves feature quoting/formatting and adds Miri strict-provenance coverage for HashPrune kernels.

Comments suppressed due to low confidence (1)

diskann-disk/src/build/configuration/build_algorithm.rs:157

If PiPNNParameters::default() is changed to keep hash_prune opt-in, this serde-defaults test should be updated to expect None instead of Some(HashPruneParameters::default()).

        assert_eq!(config.hash_prune, Some(HashPruneParameters::default()));

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+libc = "0.2"
+parking_lot = "0.12"


 impl Default for PiPNNParameters {
    fn default() -> Self {
        Self {
            c_max: 256,
            c_min: 16,
            p_samp: 0.005,
            fanout: vec![8, 3],
            k: 2,
            replicas: 1,
+            hash_prune: Some(HashPruneParameters::default()),
        }
    }


Copilot

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (1)

diskann-disk/src/build/configuration/build_algorithm.rs:82

PiPNNParameters is #[serde(default)], so deserializing PiPNN configs that omit the new hash_prune field will inherit this Default value. Setting hash_prune: Some(HashPruneParameters::default()) therefore enables HashPrune by default and can change behavior for existing JSON configs that previously used direct candidate merging. If HashPrune is meant to be opt-in (as described), make the default None and require explicit configuration to enable it.

impl Default for PiPNNParameters {
    fn default() -> Self {
        Self {
            c_max: 256,
            c_min: 16,
            p_samp: 0.005,
            fanout: vec![8, 3],
            k: 2,
            replicas: 1,
            hash_prune: Some(HashPruneParameters::default()),
        }

weiyaoluo (SeliMeli) · 2026-08-07T07:02:34Z

Validation count after deleting unsupported dynamic-k and unordered-leaf test paths: 440 diskann library tests pass with pipnn,testing. No .cargo/mutants.toml exists in any stack layer.

Route HashPrune integration through diskann::graph::pipnn, remove obsolete Cargo benchmarks, and consolidate tests around exact conversion, reservoir, concurrency, and extraction contracts.

Use the shared diskann IAI target for direct, HashPrune plus final prune, and nearest-only paths. Cover the empty-edge scratch fast path and Rust 2024 Windows FFI declaration.

Preserve legacy direct-candidate behavior when hash_prune is omitted, and reject capacity below graph degree before quantizer training or artifact creation.

Run strict-provenance Miri on dispatched hash and reservoir mutation paths, and keep libc target-specific to Linux.

Adapt HashPrune to current main, fix its concurrent aliasing boundaries, and keep all private/composition tests beside their implementation.

Use the blocking CI job as the sole HashPrune strict-provenance entry point; leave nightly workflow unchanged.

State allocation, ownership, lock, initialization, and pointer-range facts at each HashPrune unsafe boundary instead of relying on an implicit outer caller contract.

Use direct VectorRepr sketch construction, Option for optional HashPrune, raw-only slab access, and one-step nearest-ID extraction.

weiyaoluo (SeliMeli) mentioned this pull request Jul 29, 2026

PiPNN 5/6: add dedicated benchmark pipelines #1294

Open

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch 2 times, most recently from d4c2ba6 to 93cf436 Compare July 30, 2026 08:26

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch from 93cf436 to efedce4 Compare July 30, 2026 08:55

Copilot AI lite review requested due to automatic review settings July 30, 2026 11:26

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch from efedce4 to f8f27bf Compare July 30, 2026 11:26

weiyaoluo (SeliMeli) requested a review from a team July 30, 2026 11:26

Copilot started reviewing on behalf of weiyaoluo (SeliMeli) July 30, 2026 11:27 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 30, 2026 13:25

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch from f8f27bf to 661365c Compare July 30, 2026 13:25

Copilot started reviewing on behalf of weiyaoluo (SeliMeli) July 30, 2026 13:26 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 31, 2026 04:24

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch from 661365c to 25ab02b Compare July 31, 2026 04:24

Copilot started reviewing on behalf of weiyaoluo (SeliMeli) July 31, 2026 04:24 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 3, 2026 02:18

weiyaoluo (SeliMeli) force-pushed the pipnn-stack/06-hash-prune branch from 25ab02b to 10bec9e Compare August 3, 2026 02:18

Copilot started reviewing on behalf of weiyaoluo (SeliMeli) August 3, 2026 02:18 View session

weiyaoluo (SeliMeli) added 29 commits August 7, 2026 15:49

pipnn: add HashPrune candidate merging

6ed3673

build: configure HashPrune integration

de9c287

refactor(pipnn): dispatch HashPrune through diskann-wide

bbd1ef9

perf(pipnn): move unique HashPrune rows into adjacency

4c7ea05

fix(pipnn): validate HashPrune capacity against graph degree

b1c4475

ci: run Miri on HashPrune pointer kernels

2f07e14

refactor(pipnn): scope extraction scratch to Rayon jobs

482af91

fix(pipnn): validate HashPrune hash-space capacity

11174da

perf(pipnn): retain HashPrune sketch scratch capacity

3659621

docs(pipnn): map HashPrune reservoir flow

8546cc2

test(pipnn): exercise sketch scratch reuse

062279a

test(disk): adapt PiPNN adapter mismatch case

f4ad3eb

fix(pipnn): make HashPrune history independent

269cb11

docs(pipnn): name point reservoirs

0545bc0

refactor(pipnn): finish in-crate HashPrune move

4db9898

Route HashPrune integration through diskann::graph::pipnn, remove obsolete Cargo benchmarks, and consolidate tests around exact conversion, reservoir, concurrency, and extraction contracts.

bench(pipnn): add IAI candidate-merge scenarios

c5fff9d

Use the shared diskann IAI target for direct, HashPrune plus final prune, and nearest-only paths. Cover the empty-edge scratch fast path and Rust 2024 Windows FFI declaration.

fix(pipnn): keep HashPrune explicitly opt-in

9e9e2b5

Preserve legacy direct-candidate behavior when hash_prune is omitted, and reject capacity below graph degree before quantizer training or artifact creation.

ci(pipnn): gate unsafe pointer boundaries

1116c2e

Run strict-provenance Miri on dispatched hash and reservoir mutation paths, and keep libc target-specific to Linux.

test(pipnn): colocate HashPrune coverage

d9169a9

Adapt HashPrune to current main, fix its concurrent aliasing boundaries, and keep all private/composition tests beside their implementation.

fix(pipnn): satisfy Windows FFI lint

d9f28da

ci(pipnn): keep one Miri gate

9946a3f

Use the blocking CI job as the sole HashPrune strict-provenance entry point; leave nightly workflow unchanged.

docs(pipnn): make unsafe proofs local

ca50bba

State allocation, ownership, lock, initialization, and pointer-range facts at each HashPrune unsafe boundary instead of relying on an implicit outer caller contract.

refactor(pipnn): remove unused wrapper layers

15df4ea

Use direct VectorRepr sketch construction, Option for optional HashPrune, raw-only slab access, and one-step nearest-ID extraction.

docs(pipnn): simplify HashPrune comments

21a9d79

fix(pipnn): reject invalid HashPrune leaf input

7bb780c

refactor(pipnn): return HashPrune ingestion errors

099aa45

docs(pipnn): state core HashPrune contracts

6e4617b

refactor(pipnn): use domain names in HashPrune flow

e1d6d52

docs(pipnn): describe HashPrune domain actions

debcbee

Navigation Menu

Uh oh!

PiPNN 6/6: add HashPrune candidate merging - #1295

PiPNN 6/6: add HashPrune candidate merging#1295
weiyaoluo (SeliMeli) wants to merge 29 commits into
pipnn-stack/05-benchmarkfrom
pipnn-stack/06-hash-prune

weiyaoluo (SeliMeli) commented Jul 29, 2026 •

edited

Loading

Uh oh!

weiyaoluo (SeliMeli) commented Jul 29, 2026

Uh oh!

weiyaoluo (SeliMeli) commented Jul 30, 2026

Uh oh!

weiyaoluo (SeliMeli) commented Jul 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

weiyaoluo (SeliMeli) commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

weiyaoluo (SeliMeli) commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Concepts

Code map

End-to-end flow

Invariants and boundaries

Review path

Test architecture

Validation

Stack relation

Uh oh!

weiyaoluo (SeliMeli) commented Jul 29, 2026

Uh oh!

weiyaoluo (SeliMeli) commented Jul 30, 2026

Direct final-stack QC

Fixed during QC

Remaining findings

Validation

Uh oh!

weiyaoluo (SeliMeli) commented Jul 30, 2026

QC follow-up

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

weiyaoluo (SeliMeli) commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

weiyaoluo (SeliMeli) commented Jul 29, 2026 •

edited

Loading